Mobile video conferencing with digital annotation

ABSTRACT

A local user of a local mobile device is allowed to participate in a video conference session with a remote user of a remote mobile device. Live video can be shared between and collaboratively digitally annotated by the local and remote users. An application can also be shared between and collaboratively digitally annotated by the local and remote users. A digital object can also be shared between and collaboratively digitally annotated by the local and remote users.

BACKGROUND

Due to factors such as economic globalization and ongoing advances incomputing, data communication, and computer networking technologies,people across the globe are becoming increasingly mobile. Examples ofsuch technology advances include the Internet, the World Wide Web,wireless local area and wide area networks, and mobile computing devicesand applications. Various types of mobile computing devices are nowcommercially available which allow people to affordably and convenientlyperform full-fledged computing and data communication activities whilethey are on the move. Smartphones and tablet computers are two examplesof such devices. The popularity of mobile computing devices is growingrapidly, as is the types of mobile computing applications that areavailable. Accordingly, the number of people that regularly uses amobile computing device to perform their online computing,communication, and information management tasks is growing rapidly. Infact, mobile computing devices have become a principal computing devicefor many people.

In addition to people becoming increasingly mobile, businesses andfamilies are becoming increasingly geographically distributed. As aresult, people in different geographic locations have an increasingdesire to collaborate and exchange information in a live (i.e., dynamicand interactive) manner without having to spend the time, money andeffort associated with traveling to a common geographic location. Videoconferencing is a popular technique which provides for the live exchangeof both video and audio information between two or more people who aresituated remotely from one another and are linked by a datacommunication network. Video conferencing thus allows the people tocollaborate and exchange information in a live manner.

SUMMARY

This Summary is provided to introduce a selection of concepts, in asimplified form, that are further described hereafter in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Mobile video conferencing technique embodiments described hereingenerally allow a local user of a local mobile device to participate ina video conference session with a remote user of a remote mobile device,where the local and remote mobile devices are interconnected by anetwork, and the local mobile device includes a display screen, a frontvideo capture device that is pointed toward the local user and captureslive video of the local user, and a rear video capture device that ispointed away from the local user and captures live video of a scene thatis in front of the local user. In one exemplary embodiment live video isreceived over the network from the remote mobile device and is displayedon the entire display screen. Upon the local user selecting either thelive video of the local user, or the live video of the scene that is infront of the local user, the selected live video is transmitted over thenetwork to the remote mobile device, and is also resized as necessary tofit within an inset sector on the display screen. The resized selectedlive video is then displayed within the inset sector. Whenever the localuser makes a first digital annotation on the received live video, thefirst digital annotation is displayed on top of the received live video,and the first digital annotation is transmitted over the network to theremote mobile device. Whenever the remote user makes a second digitalannotation on the selected live video, the second digital annotation isreceived over the network from the remote mobile device, the receivedsecond digital annotation is resized as necessary to fit within theinset sector, and the resized received second digital annotation isdisplayed on top of the resized selected live video.

In another exemplary embodiment the live video of the local user istransmitted over the network to the remote mobile device. Live video ofthe remote user is received over the network from the remote mobiledevice, the received live video is resized as necessary to fit withinthe inset sector on the display screen, and the resized received livevideo is displayed within the inset sector. Upon the local user openingan application on the entire display screen, a message to run theapplication is transmitted over the network to the remote mobile device.Whenever the local user makes a first digital annotation on theapplication, the first digital annotation is displayed on top of theapplication, and the first digital annotation is transmitted over thenetwork to the remote mobile device. Whenever the remote user makes asecond digital annotation on the application, the second digitalannotation is received over the network from the remote mobile device,and the received second digital annotation is displayed on top of theapplication.

In yet another exemplary embodiment the live video of the local user istransmitted over the network to the remote mobile device. Live video ofthe remote user is received over the network from the remote mobiledevice, the received live video is resized as necessary to fit withinthe inset sector on the display screen, and the resized received livevideo is displayed within the inset sector. Upon the local userdisplaying a digital object on the entire display screen, the object istransmitted over the network to the remote mobile device. Whenever thelocal user makes a first digital annotation on the object, the firstdigital annotation is displayed on top of the object, and the firstdigital annotation is transmitted over the network to the remote mobiledevice. Whenever the remote user makes a second digital annotation onthe object, the second digital annotation is received over the networkfrom the remote mobile device, and the received second digitalannotation is displayed on top of the object.

DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the mobile videoconferencing technique embodiments described herein will become betterunderstood with regard to the following description, appended claims,and accompanying drawings where:

FIG. 1 is a diagram illustrating an exemplary embodiment, in simplifiedform, of an architectural framework for implementing the mobile videoconferencing technique embodiments described herein.

FIG. 2 is a diagram illustrating one embodiment, in simplified form, ofa video sharing feature and a collaborative digital annotation featureof the conferencing technique embodiments described herein.

FIG. 3 is a diagram illustrating an exemplary embodiment, in simplifiedform, of a view sharing feature of the conferencing techniqueembodiments described herein.

FIG. 4 is a flow diagram illustrating one embodiment, in simplifiedform, of a process for allowing a local user of a local mobile device toparticipate in a video conference session with a remote user of a remotemobile device.

FIG. 5 is a flow diagram illustrating another embodiment, in simplifiedform, of a process for allowing the local user of the local mobiledevice to participate in a video conference session with the remote userof the remote mobile device.

FIG. 6 is a flow diagram illustrating yet another embodiment, insimplified form, of a process for allowing the local user of the localmobile device to participate in a video conference session with theremote user of the remote mobile device.

FIG. 7 is a diagram illustrating a simplified example of ageneral-purpose computer system on which various embodiments andelements of the mobile video conferencing technique, as describedherein, may be implemented.

FIG. 8 is a diagram illustrating another embodiment, in simplified form,of the video sharing feature and the collaborative digital annotationfeature of the conferencing technique embodiments described herein.

DETAILED DESCRIPTION

In the following description of mobile video conferencing techniqueembodiments (hereafter simply referred to as conferencing techniqueembodiments) reference is made to the accompanying drawings which form apart hereof, and in which are shown, by way of illustration, specificembodiments in which the conferencing technique can be practiced. It isunderstood that other embodiments can be utilized and structural changescan be made without departing from the scope of the conferencingtechnique embodiments.

It is also noted that for the sake of clarity specific terminology willbe resorted to in describing the conferencing technique embodimentsdescribed herein and it is not intended for these embodiments to belimited to the specific terms so chosen. Furthermore, it is to beunderstood that each specific term includes all its technicalequivalents that operate in a broadly similar manner to achieve asimilar purpose. Reference herein to “one embodiment”, or “anotherembodiment”, or an “exemplary embodiment”, or an “alternate embodiment”,or “one implementation”, or “another implementation”, or an “exemplaryimplementation”, or an “alternate implementation” means that aparticular feature, a particular structure, or particularcharacteristics described in connection with the embodiment orimplementation can be included in at least one embodiment of theconferencing technique. The appearances of the phrases “in oneembodiment”, “in another embodiment”, “in an exemplary embodiment”, “inan alternate embodiment”, “in one implementation”, “in anotherimplementation”, “in an exemplary implementation”, and “in an alternateimplementation” in various places in the specification are notnecessarily all referring to the same embodiment or implementation, norare separate or alternative embodiments/implementations mutuallyexclusive of other embodiments/implementations. Yet furthermore, theorder of process flow representing one or more embodiments orimplementations of the conferencing technique does not inherentlyindicate any particular order not imply any limitations of theconferencing technique.

The term “screen-contacting gesture” is used herein to refer to either aphysical tap or stroke that is made directly on a touch-sensitivedisplay screen of a computing device by a user via a given user inputmodality. The term “mobile device” is used herein to refer to anetworking-enabled, handheld, client computing device that is carried bya user and can run various mobile computing applications including, butnot limited to, ones that enable Internet access. As is appreciated inthe art of mobile computing technology, today's mobile devices includean integrated touch-sensitive display screen that can detect thepresence, location, and path of movement if applicable, ofscreen-contacting gestures that the user makes on the display screen.Many of today's mobile devices also include an integrated front videocapture device that is pointed toward the user, and may further includean integrated rear video capture device that is pointed away from theuser. The latest generations of smartphones and tablet computers are buttwo examples of mobile devices that include an integratedtouch-sensitive display screen, front video capture device, and rearvideo capture device. For simplicity sake, the touch-sensitive displayscreen of a mobile device is hereafter simply referred to as a displayscreen.

The term “sector” is used herein to refer to a segmented region of adisplay screen in which a particular type of graphical user interfaceand/or information is displayed, or a particular type of function isperformed by a user. The term “inset sector” is used herein to refer toa sector on the display screen within which secondary information (suchas live video that is currently being either received from ortransmitted to a remote user, among other things) is displayed. In oneembodiment of the conferencing technique described herein the insetsector is implemented as a picture-in-picture element on the displayscreen. In another embodiment of the conferencing technique the insetsector is implemented as a semi-transparent overlay layer that fills theentire display screen.

1.0 Mobile Video Conferencing with Digital Annotation

Generally speaking, the conferencing technique embodiments describedherein allow a local user of a local mobile device to participate in avideo conference session with a remote user of a remote mobile device.The conferencing technique embodiments support a variety of differentfeatures which can be implemented either independently or in anycombination. Various examples of these features are described in moredetail hereafter.

The conferencing technique embodiments described herein are advantageousfor various reasons including, but not limited to, the following. Theconferencing technique embodiments generally provide for thebidirectional sharing of various items of digital information such aslive video, snapshots thereof, digital objects, and applications, amongothers (hereafter sometimes simply referred to as items of digitalinformation that are currently being shared) between the local andremote users participating in the video conference session. Theconferencing technique embodiments also allow the users to participatein the session in an efficient and affordable manner regardless of wherein the world they each happen to be located. In other words, the usersdo not have to spend the time and money associated with finding andutilizing a public video conference service and/or related facility inorder to be able to participate in the session.

The conferencing technique embodiments described herein are alsoadvantageous in that they allow the users to share and discuss theiractivities with each other in a wide variety of contexts andenvironments (e.g., outdoors, in neighborhoods, at sporting events, andat child-related events such as play activities and birthday parties,among others). In other words, the conferencing technique embodimentsallow the users to flexibly communicate and collaborate with each othervia video in almost any environment. The conferencing techniqueembodiments are also easy to deploy and easy to use. The conferencingtechnique embodiments are also compatible with various conventionalmobile devices and various conventional data communication networks.

The conferencing technique embodiments described herein are alsoadvantageous in that they generally optimize the quality, effectivenessand productivity of communication and collaborative interaction betweenthe local and remote users participating in the video conferencesession. More particularly and as will be described in more detailhereafter, the conferencing technique embodiments allow virtualface-to-face interaction between the users at the same time theyparticipate in other collaborative activities such as collaborativelyviewing, manipulating and digitally annotating a given item of digitalinformation that is currently being shared between the users. Theconferencing technique embodiments also allow the users to freely moveabout while they are participating in the session (as opposed to havingtheir physical location be limited to a position directly in front of anon-mobile/stationary computing device). As a result, the conferencingtechnique embodiments create a rich and satisfying video conferenceexperience for the users.

1.1 Architectural Framework

FIG. 1 illustrates an exemplary embodiment, in simplified form, of anarchitectural framework for implementing the conferencing techniqueembodiments described herein. Generally speaking and as exemplified inFIG. 1, the conferencing technique embodiments allow a local user 100 ofa local mobile device 102 to participate in a video conference sessionwith a remote user 104 of a remote mobile device 106. The local andremote mobile devices and are interconnected by a distributed datacommunication network 108 (hereafter simply referred to as the network).

In the particular embodiment exemplified in FIG. 1, both the local andremote mobile devices 102 and 106 are connected to the network 108 viaconventional wireless connections 110 and 112. As is appreciated in theart of communication networks, the network can be either a publiccommunication network such as the Internet (among others), or a privatecommunication network such as an intranet (among others). A givenwireless connection 110 or 112 can be implemented in various waysdepending on the particular type of mobile device 102 or 106 that isbeing used by its user 100 or 104 and the types of wireless networkservice that are currently available in the particular location wherethe user happens to be situated during the video conference session. Byway of example but not limitation, the wireless connection 110 or 112can be a conventional Wi-Fi local area network (LAN) connection to aconventional Wi-Fi access point device (not shown). The wirelessconnection 110 or 112 can also be a conventional cellular wide areanetwork (WAN) connection which supports one or more differentconventional mobile telecommunication data services such as GPRS(general packet radio service—also known as 2.5G), EDGE (enhanced datarates for GSM (global system for mobile communications) evolution—alsoknown as 2.75G), 3G (third generation), and 4G (fourth generation),among others.

Referring again to FIG. 1, the local mobile device 102 includes variousfunctional components which are integrated there-within. Examples ofthese functional components include, but are not limited to, a localdisplay screen 114, a local front video capture device 116 (such as acompact video camera, or the like) that is pointed toward 120 the localuser 100, and a local rear video capture device 118 (such as a compactvideo camera, or the like) that is pointed away 122 from the local user.Accordingly, the local front and rear video capture devices 116 and 118are pointed in substantially opposite directions 120 and 122 such thatthe local front video capture device captures a live video stream(hereafter simply referred to as live video) of whatever issubstantially in front of the local display screen, and the local rearvideo capture device captures live video of whatever is substantiallybehind the local display screen. The local mobile device can optionallyalso include additional functional components which are integratedthere-within such as a local audio capture device (not shown) (such as acompact microphone, or the like), a local audio output device (notshown) (such as one or more compact loudspeakers, or the like), andfunctionality that enables the computing device to operate as atelephone. The local audio capture device captures live audio emanatingfrom the local user and their surrounding environment.

Referring again to FIG. 1, the remote mobile device 106 similarlyincludes various functional components which are integratedthere-within. Examples of these functional components include, but arenot limited to, a remote display screen 124, a remote front videocapture device 126 (such as a compact video camera, or the like) that ispointed toward 128 the remote user 104, and a remote rear video capturedevice 130 (such as a compact video camera, or the like) that is pointedaway 132 from the remote user. Accordingly, the remote front and rearvideo capture devices 126 and 130 are pointed in substantially oppositedirections 128 and 132 such that the remote front video capture devicecaptures live video of whatever is substantially in front of the remotedisplay screen, and the remote rear video capture device captures livevideo of whatever is substantially behind the remote display screen. Theremote mobile device can optionally also include additional functionalcomponents which are integrated there-within such as a remote audiocapture device (now shown) (such as a compact microphone, or the like),a remote audio output device (not shown) (such as one or more compactloudspeakers, or the like), and functionality that enables the computingdevice to operate as a telephone. The remote audio capture devicecaptures live audio emanating from the remote user and their surroundingenvironment.

Referring again to FIG. 1, during a video conference session the localuser 100 will generally either be holding the local mobile device 102 inone of their hands, or will place this device on a table top in front ofthem. In either case, the local user will generally be looking at thelocal display screen 114 during the video conference session so that thelocal front video capture device 116 will generally capture live videoof the local user (e.g., their head and torso). As such, the live videoof the local user can include various types of information such as thelocal user's eye gaze direction, facial expressions, body language, handgestures, other body gestures, and the like. Whenever the local user isholding the local mobile device in one of their hands, the local rearvideo capture device 118 will generally capture live video of the scenethat is substantially in front of the local user (i.e., whatever thelocal user may currently be looking at). Similarly, the remote user 104will generally either be holding the remote mobile device 106 in one oftheir hands, or will place this device on a table top in front of them.In either case, the remote user will generally be looking at the remotedisplay screen 124 during the video conference session so that theremote front video capture device 126 will generally capture live videoof the remote user (e.g., their head and torso). As such, the live videoof the remote user can include various types of information such as theremote user's eye gaze direction, facial expressions, body language,hand gestures, other body gestures, and the like. In another situationwhere the remote user is holding the remote mobile device in one oftheir hands, the remote rear video capture device 130 will generallycapture live video of the scene that is substantially in front of theremote user (i.e., whatever the remote user may currently be lookingat).

1.2 Video Sharing and Collaborative Digital Annotation Features

This section describes a video sharing feature and a collaborativedigital annotation feature of the conferencing technique embodimentsdescribed herein.

FIG. 2 illustrates one embodiment, in simplified form, of the videosharing feature and the collaborative digital annotation feature of theconferencing technique embodiments described herein. As exemplified inFIG. 2, during a video conference session the local user 200 can chooseto send either the live video that is currently being captured by thelocal front video capture device 216 or the live video that is currentlybeing captured by the local rear video capture device 218 to the remoteuser 204, and the local mobile device 202 will transmit the chosen livevideo (shown as Video 1) (hereafter sometimes referred to as theselected live video), along with the live audio that is currently beingcaptured by the local audio capture device, over the network 208 to theremote mobile device 206. Similarly, the remote user can choose to sendeither the live video that is currently being captured by the remotefront video capture device 226 or the live video that is currently beingcaptured by the remote rear video capture device 230 to the local user,and the remote mobile device will transmit the chosen live video (shownas Video 2), along with the live audio that is currently being capturedby the remote audio capture device, over the network to the local mobiledevice.

Referring again to FIG. 2, the local display screen 214 is configured inwhat is known as a picture-in-picture format where the live video thatthe local mobile device 202 receives over the network 208 from theremote mobile device 206 (i.e., Video 2) is displayed on the entirelocal display screen, the live video that the local mobile device iscurrently transmitting to the remote mobile device (i.e., Video 1) isresized as necessary to fit within an inset sector 234 on the localdisplay screen, and this resized live video (shown as Video 1R) isdisplayed within this sector 234 (i.e., this sector 234 shows the localuser 200 a preview of the live video they have chosen to send to theremote user 204). Similarly, the remote display screen 224 is alsoconfigured in a picture-in-picture format where the live video that theremote mobile device receives over the network from the local mobiledevice (i.e., Video 1) is displayed on the entire remote display screen,the live video that the remote mobile device is currently transmittingto the local mobile device (i.e., Video 2) is resized as necessary tofit within an inset sector 236 on the remote display screen, and thisresized live video (shown as Video 2R) is displayed within this sector236 (i.e., this sector 236 shows the remote user a preview of the livevideo they have chosen to send to the local user).

Referring again to FIG. 2, whenever the local mobile device 202 isdisplaying the live video (Video 2) it receives over the network 208from the remote mobile device 206, the local mobile device can audiblyreproduce the live audio it also receives over the network from theremote mobile device using the local audio output device. Similarly,whenever the remote mobile device is displaying the live video (Video 1)it receives over the network from the local mobile device, the remotemobile device can audibly reproduce the live audio it also receives overthe network from the local mobile device using the remote audio outputdevice. It will be appreciated that the live video and live audio agiven mobile device receives over the network can be managedindependently. More particularly and by way of example but notlimitation, in a situation where the local user 200 minimizes the livevideo (Video 2) so that it is no longer being displayed on the entirelocal display screen 214, the local mobile device can continue toaudibly reproduce the live audio that is associated with this live video(e.g., the local user can continue to hear the remote user's 204 voiceeven though their video (Video 2) is not longer being displayed).Additionally, the local user can choose to mute this live audio at anytime, regardless of whether or not it associated live video is beingdisplayed).

Generally speaking and referring again to FIG. 2, the conferencingtechnique embodiments described herein allow both the local user 200 andthe remote user 204 to collaboratively digitally annotate (i.e., mark-upwith digital ink) the live video they are receiving over the networkfrom each other, where any such annotations made by one user areremotely visible to the other user in an appropriately resized andrelocated form very shortly after the annotations are made. Moreparticularly, whenever the local user makes a digital annotation 238 onthe live video (Video 2) that is currently being displayed on the entirelocal display screen 214 (i.e., the live video the remote user haschosen to send to the local user), the local mobile device 202 willdisplay the annotation 238 on top of the live video (Video 2) that iscurrently being displayed on the entire local display screen, and willalso transmit the annotation over the network 208 to the remote mobiledevice 206. Upon receiving this digital annotation over the network fromthe local mobile device, the remote mobile device will resize thereceived annotation as necessary to fit within the inset sector 236 onthe remote display screen 224, and will then display the resizedannotation 240 on top of the live video (Video 2R) that is currentlybeing displayed within this sector 236 (i.e., the preview of the livevideo the remote user has chosen to send to the local user). In otherwords and as exemplified in FIG. 2, the location and size of the digitalannotation 238 that is made by the local user is mapped from the localdisplay screen to the inset sector 236 on the remote display screen.

Similarly and referring again to FIG. 2, whenever the remote user 204makes a digital annotation 242 on the live video (Video 1) that iscurrently being displayed on the entire remote display screen 224 (i.e.,the live video the local user 200 has chosen to send to the remoteuser), the remote mobile device 206 will display the annotation 242 ontop of the live video (Video 1) that is currently being displayed on theentire remote display screen, and will also transmit the annotation overthe network 208 to the local mobile device 202. Upon receiving thisdigital annotation over the network from the remote mobile device, thelocal mobile device will resize the received annotation as necessary tofit within the inset sector 234 on the local display screen 214, andwill then display the resized annotation 244 on top of the live video(Video 1R) that is currently being displayed within this sector 234(i.e., the preview of the live video the local user has chosen to sendto the remote user). In other words and as exemplified in FIG. 2, thelocation and size of the digital annotation 242 that is made by theremote user is mapped from the remote display screen to the inset sector234 on the local display screen.

Generally speaking and referring again to FIG. 2, the conferencingtechnique embodiments described herein also allow both the local user200 and the remote user 204 to collaboratively digitally annotate thelive video they have chosen to send to each other, where any suchannotations made by one user are remotely visible to the other user inan appropriately resized and relocated form very shortly after theannotations are made. More particularly, whenever the local user makes adigital annotation 244 on the live video (Video 1R) that is currentlybeing displayed within the inset sector 234 on the local display screen214 (i.e., the preview of the live video the local user has chosen tosend to the remote user), the local mobile device 202 will display theannotation 244 on top of the live video (Video 1R) that is currentlybeing displayed within this sector 234, and will also transmit theannotation over the network 208 to the remote mobile device 206. Uponreceiving this digital annotation over the network from the local mobiledevice, the remote mobile device will resize the received annotation asnecessary to fit on the entire remote display screen 224, and will thendisplay the resized annotation 242 on top of the live video (Video 1)that is currently being displayed on the entire remote display screen.In other words and as exemplified in FIG. 2, the location and size ofthe digital annotation 244 that is made by the local user is mapped fromthe inset sector 234 on the local display screen to the remote displayscreen.

Similarly and referring again to FIG. 2, whenever the remote user 204makes a digital annotation 240 on the live video (Video 2R) that iscurrently being displayed within the inset sector 236 on the remotedisplay screen 224 (i.e., the preview of the live video the remote userhas chosen to send to the local user 200), the remote mobile device 206will display the annotation 240 on top of the live video (Video 2R) thatis currently being displayed within this sector 236, and will alsotransmit the annotation over the network 208 to the local mobile device202. Upon receiving this digital annotation over the network from theremote mobile device, the local mobile device will resize the receivedannotation as necessary to fit on the entire local display screen 214,and will then display the resized annotation 238 on top of the livevideo (Video 2) that is currently being displayed on the entire localdisplay screen. In other words and as exemplified in FIG. 2, thelocation and size of the digital annotation 240 that is made by theremote user is mapped from the inset sector 236 on the remote displayscreen to the local display screen.

As is appreciated in the art of data communications, time lag isinherent in the network that interconnects the local and remote mobiledevices and jitter may also be present in this network. Accordingly, itis possible for a given digital annotation that is made by one user on agiven live video that is currently being displayed on one mobile deviceto be received over the network by another mobile device before theframes of the live video on which the annotation was made are received.In order to account for the time lag and any jitter that is present inthe network, the conferencing technique embodiments described herein cantime correlate each digital annotation that the local and remote usersmake to the particular frames of the particular live video upon whichthe annotation is made. As a result of this time correlation, a givendigital annotation that is received over the network will not bedisplayed until the particular frames of the particular live video uponwhich the annotation was made are received, rendered and displayed.

FIG. 8 illustrates another embodiment, in simplified form, of the videosharing feature and the collaborative digital annotation feature of theconferencing technique embodiments described herein. As exemplified inFIG. 8, during a video conference session the local user 800 can chooseto display the live video that is currently being captured by the localrear video capture device 818 (which is shown as Video 1) on the entirelocal display screen 814, and also choose to send this live video to theremote user 804. The local mobile device 802 will display this livevideo (Video 1) on the entire local display screen 814, and will alsotransmit this live video, along with the live audio that is currentlybeing captured by the local audio capture device, over the network 808to the remote mobile device 806. When the remote mobile device 806receives this live video (Video 1) over the network 808 from the localmobile device 802, the remote mobile device will display this live videoon the entire remote display screen 824. Accordingly, in this particularembodiment both the local and remote users 800 and 804 will be viewingthe same live video (namely Video 1) on their respective entire displayscreens 814 and 824 and the users can collaboratively digitally annotatethis video as follows.

Referring again to FIG. 8, whenever the local user 800 makes a digitalannotation 838 on the live video (Video 1) that is currently beingdisplayed on the entire local display screen 814 (i.e., the live videothe local user has chosen to send to the remote user 804), the localmobile device 802 will display the annotation 838 on top of the livevideo (Video 1) that is currently being displayed on the entire localdisplay screen, and will also transmit the annotation over the network808 to the remote mobile device 806. Upon receiving this digitalannotation over the network 808 from the local mobile device 802, theremote mobile device 806 will display the received annotation 822,without it being resized or repositioned, on top of the live video(Video 1) that is currently being displayed on the entire remote displayscreen 824. Similarly, whenever the remote user 804 makes a digitalannotation 842 on the live video (Video 1) that is currently beingdisplayed on the entire remote display screen 824, the remote mobiledevice 806 will display the annotation 842 on top of the live video(Video 1) that is currently being displayed on the entire remote displayscreen 824, and will also transmit the annotation over the network 808to the local mobile device 802. Upon receiving this digital annotationover the network 808 from the remote mobile device 806, the local mobiledevice 802 will display the received annotation 820, without it beingresized or repositioned, on top of the live video (Video 1) that iscurrently being displayed on the entire local display screen 814.

Referring again to FIG. 8, the local mobile device 802 will alsotransmit the live video that is currently being captured by the localfront video capture device 816 over the network 808 to the remote mobiledevice 806. Upon receiving this live video from the local mobile device802, the remote mobile device 806 will resize it as necessary to fitwithin an inset sector 836 on the remote display screen 824, and willdisplay this resized live video within this sector 836. Similarly, theremote mobile device 806 will also transmit the live video that iscurrently being captured by the remote front video capture device 826over the network 808 to the local mobile device 802. Upon receiving thislive video from the remote mobile device 806, the local mobile device802 will resize it as necessary to fit within an inset sector 834 on thelocal display screen 814, and will display this resized live videowithin this sector 834.

Referring again to FIG. 8, the local mobile device 802 can optionallyresize the live video that is currently being captured by the localfront video capture device 816 as necessary to fit within another insetsector 844 on the local display screen 814, and display this resizedlive video within this sector 844 (thus providing the local user 800with a preview of the live video that is currently being captured by thelocal front video capture device 816). Similarly, the remote mobiledevice 806 can optionally resize the live video that is currently beingcaptured by the remote front video capture device 826 as necessary tofit within another inset sector 840 on the remote display screen 824,and display this resized live video within this sector 840 (thusproviding the remote user 806 with a preview of the live video that iscurrently being captured by the remote front video capture device 826).

1.3 User-Modifiable Display Screen Configuration Features

A given user of a given mobile device can modify the configuration oftheir mobile device's display screen in various ways to meet their needsand optimize an interaction they are currently having with the otheruser during a video conference session. Examples of such ways include,but are not limited to, an inset sector repositioning feature, an insetsector resizing feature, and a view sharing feature, each of which willnow be described in more detail.

Referring again to FIG. 2, the inset sector repositioning and resizingfeatures of the conferencing technique embodiments described hereingenerally operate in the following manner. The local user 200 canreposition the inset sector 234 on the local display screen 214.Whenever the inset sector 234 is repositioned on the local displayscreen, the live video (Video 1R) and any digital annotations 244 whichare currently being displayed there-within are repositioned accordingly.The local user can also resize the inset sector 234 on the local displayscreen. Whenever the inset sector 234 is resized on the local displayscreen, the live video (Video 1R) and any digital annotations 244 whichare currently being displayed there-within are resized accordingly. Theremote user 204 can similarly either reposition or resize the insetsector 236 on the remote display screen 224, and the live video (Video2R) and any digital annotations 240 which are currently being displayedthere-within will be either repositioned or resized accordingly. It willbe appreciated that in addition to resizing and repositioning the insetsector, a given user can also close the inset sector so that it iscompletely removed from the display screen of their mobile device. Agiven user can also swap the live video and any digital annotationswhich are currently being displayed within their display screen's insetsector and the live video any digital annotations which are currentlybeing displayed on their entire display screen (e.g., the local user 200can move the Video 1R and its annotation 244 from the inset sector 234to the entire local display screen 214, and move the Video 2 and itsannotation 238 from the entire local display screen 214 to the insetsector 234).

FIG. 3 illustrates an exemplary embodiment, in simplified form, of theview sharing feature of the conferencing technique embodiments describedherein. Generally speaking, the view sharing feature allows a given userto choose which live video is displayed on the entire display screen oftheir mobile device by using the just-described swap feature, amongother ways. As a result, both the local and remote users 300 and 304 canview a common live video on their entire local and remote displayscreens 314 and 324 and collaboratively digitally annotate this commonlive video. In the particular embodiment exemplified in FIG. 3, wheneverthe local user 300 chooses to display the live video that they havechosen to send to the remote user 304 (i.e., the live video that iscurrently being captured by either the local front video capture device316 or the local rear video capture device 318) (which is shown as Video1), the local mobile device 302 will display the live video that theyhave chosen to send to the remote user on the entire local displayscreen 314. Accordingly, in this particular embodiment both the localand remote users will be viewing the same live video (namely Video 1) ontheir respective entire display screens and the users cancollaboratively digitally annotate this video. More particularly and asexemplified in FIG. 3, whenever the local user makes a digitalannotation 338 on the live video (Video 1) that is currently beingdisplayed on the entire local display screen (i.e., the live video thelocal user has chosen to send to the remote user), the local mobiledevice will display the annotation 338 on top of the live video(Video 1) that is currently being displayed on the entire local displayscreen, and will also transmit the annotation over the network 308 tothe remote mobile device 306. Upon receiving this digital annotationover the network from the local mobile device, the remote mobile devicewill display the received annotation 310, without it being resized orrepositioned, on top of the live video (Video 1) that is currently beingdisplayed on the entire remote display screen 324.

Similarly and referring again to FIG. 3, whenever the remote user 304makes a digital annotation 342 on the live video (Video 1) that iscurrently being displayed on the entire remote display screen 324 (i.e.,the live video the local user 300 has chosen to send to the remoteuser), the remote mobile device 306 will display the annotation 342 ontop of the live video (Video 1) that is currently being displayed on theentire remote display screen, and will also transmit the annotation overthe network 308 to the local mobile device 302. Upon receiving thisdigital annotation over the network from the remote mobile device, thelocal mobile device will display the received annotation 312, without itbeing resized or repositioned, on top of the live video (Video 1) thatis currently being displayed on the entire local display screen 314.

Referring again to FIG. 3, it will be appreciated that an alternateembodiment (not shown) of the view sharing feature is also possiblewhere, rather than the local user 300 changing the live video that iscurrently being displayed on the entire local display screen 314 as justdescribed, the remote user 304 can similarly change the live video thatis currently being displayed on the entire remote display screen 324 tobe the live video that they have chosen to send to the local user (i.e.,the live video that is currently being captured by either the remotefront video capture device 326 or the remote rear video capture device330). Accordingly, in this particular embodiment both the local andremote users will also be viewing the same live video (namely the livevideo that is currently being captured by either the remote front videocapture device or the remote rear video capture device) on theirrespective entire display screens and the users can collaborativelydigitally annotate this video in the just described manner.

It will also be appreciated that the view sharing feature isadvantageous in situations where the local user is helping the remoteuser do something, or vice versa. By way of example but not limitation,consider a situation where a son and his mother are using theconferencing technique embodiments described herein to participate in avideo conference session during which the son is helping his mother hookup a new high-end television she just purchased. The mother can pointthe rear video capture device on her mobile device at the block ofconnectors on the back of the television and all of the dangling cablesto be attached thereto, and can then change the live video that iscurrently being displayed on the entire display screen on her mobiledevice to be the live video that is currently being captured by thisrear video capture device (i.e., the live video of these connectors anddangling cables). Upon receiving and viewing this live video of theconnectors and dangling cables on the entire display screen on hismobile device, the son can digitally annotate this video withinformation showing the mother which cables are to be attached to whichconnectors, and the mother can see his annotations on the video that isbeing displayed on her screen. The son and mother can also speak to eachother and see each other via the aforementioned inset sectors on theirrespective display screens.

1.4 Video Snapshot Feature

This section describes a video snapshot feature of the conferencingtechnique embodiments described herein. The video snapshot featuregenerally allows both the local and remote users to view a common livevideo snapshot on their entire local and remote display screens andcollaboratively digitally annotate this live video snapshot. Moreparticularly, in an exemplary embodiment of the video snapshot featurethe local user can take a snapshot of the live video they have chosen tosend to the remote user. Whenever the local user takes such a snapshot,the local mobile device will display the snapshot on the entire localdisplay screen, and will also transmit the snapshot over the network tothe remote mobile device. Upon receiving the snapshot over the networkfrom the local mobile device, the remote mobile device will display thesnapshot on the entire remote display screen. Accordingly, both thelocal and remote users will be viewing the same snapshot on theirrespective entire display screens and the users can collaborativelydigitally annotate this snapshot as follows.

Whenever the local user makes a digital annotation on the snapshot thatis currently being displayed on the entire local display screen, thelocal mobile device will display the annotation on top of the snapshotthat is currently being displayed on the entire local display screen,and will also transmit the annotation over the network to the remotemobile device. Upon receiving this digital annotation over the networkfrom the local mobile device, the remote mobile device will display thereceived annotation, without it being resized or repositioned, on top ofthe snapshot that is currently being displayed on the entire remotedisplay screen. Similarly, whenever the remote user makes a digitalannotation on the snapshot that is currently being displayed on theentire remote display screen, the remote mobile device will display theannotation on top of the snapshot that is currently being displayed onthe entire remote display screen, and will also transmit the annotationover the network to the local mobile device. Upon receiving this digitalannotation over the network from the remote mobile device, the localmobile device will display the received annotation, without it beingresized or repositioned, on top of the snapshot that is currently beingdisplayed on the entire local display screen.

It is noted that any digital annotations that are made by the local orremote users can be anchored to the snapshot that is currently beingdisplayed on the entire local and remote display screens. As a result,whenever a given user zooms in on (i.e., magnifies) the snapshot, anydigital annotations that are currently being displayed on top of thesnapshot will be magnified accordingly. Whenever the user zooms out on(i.e., shrinks/minifies) the snapshot, any digital annotations that arecurrently being displayed on top of the snapshot will be minifiedaccordingly. Whenever the user scrolls the snapshot in a desireddirection, any digital annotations that are currently being displayed ontop of the snapshot will be scrolled accordingly.

It is also noted that the just described roles of the local and remoteusers can be reversed. In other words, in an alternate embodiment of thevideo snapshot feature the remote user can take a snapshot of the livevideo that they have chosen to send to the local user. Whenever theremote user takes such a snapshot, the remote mobile device will displaythe snapshot on the entire remote display screen, and will also transmitthe snapshot over the network to the local mobile device. Upon receivingthe snapshot over the network from the remote mobile device, the localmobile device will display the snapshot on the entire local displayscreen. Both the local and remote users can then collaborativelydigitally annotate the snapshot in the just described manner.

In a default embodiment of the video snapshot feature the local mobiledevice transmits the live video that is currently being captured by thelocal front video capture device over the network to the remote mobiledevice, and upon receiving this live video the remote mobile devicedisplays it within the inset sector on the remote display screen.Similarly, the remote mobile device transmits the live video that iscurrently being captured by the remote front video capture device overthe network to the local mobile device, and upon receiving this livevideo the local mobile device displays it within the inset sector on thelocal display screen. Accordingly, in the inset sector on the localdisplay screen the local user can see live video of whatever issubstantially in front of the remote display screen (e.g., the remoteuser's head and torso), and in the inset sector on the remote displayscreen the remote user can see live video of whatever is substantiallyin front of the local display screen (e.g., the local user's head andtorso). This allows each user to see live video of the other user whilethey are collaboratively discussing and annotating a given snapshot ofthe live video of whatever is substantially behind either the local orremote display screen (e.g., the scene that is substantially in front ofeither the local or remote user).

Alternate embodiments of the video snapshot feature are also possiblewhere, as described heretofore, a given user can choose to send the livevideo that is currently being captured by their rear video capturedevice to the other user. In such a situation the given user's mobiledevice will transmit the live video that is currently being captured byits rear video capture device over the network to the other mobiledevice, and upon receiving this live video the other mobile device willdisplay it within the inset sector on the other mobile device's displayscreen. Accordingly, in this inset sector the other user can see livevideo of whatever is substantially behind the given user's displayscreen.

It will be appreciated that the video snapshot feature can be useful inthe aforementioned situation where the son is helping his mother hook upthe television she just purchased. By way of example but not limitation,the mother can take a snapshot of the live video that is currently beingcaptured by the rear video capture device on her mobile device (i.e., asnapshot of the block of connectors on the back of the television andall of the dangling cables to be attached thereto). Both the son andmother can then discuss the snapshot and annotate it as needed. It willalso be appreciated that the video snapshot feature is advantageous inthe common situation where the given user is unable to keep their mobiledevice substantially steady while they are holding it so that itsfront/rear video capture device is moving relative to the scene it iscapturing. The reason for this is that the video snapshot feature allowsany digital annotations which the user makes to maintain theirorientation relative to an object in the snapshot on top of and/oraround which the annotations were made.

1.5 Object Tracking and Image Stabilization Features

This section describes an object tracking feature and an imagestabilization feature of the conferencing technique embodimentsdescribed herein.

The object tracking feature recognizes and tracks objects that exist ina live video upon which a user is making one or more digitalannotations, and then anchors the annotations to these objects, wherethis object recognition and tracking is performed using conventionalcomputer vision and artificial intelligence methods. It will thus beappreciated that the object tracking feature is advantageous in theaforementioned situation where a user is unable to keep their mobiledevice substantially steady so that its front/rear video capture deviceis moving relative to the scene it is capturing. The reason for this isthat even if the front/rear video capture device moves or shakes whileit is capturing live video of the scene (hence causing the position ofobjects that exist in the scene to change in the live video), the objecttracking feature allows any digital annotations which the user makes totrack the objects and maintain their orientation relative to the objectsif the objects move. The object tracking feature can also be used toallow a user to annotate an object that is moving within the scene(e.g., circle a particular runner that is competing in a race at a trackmeet) and have the annotation move with the object as it moves withinthe scene.

The image stabilization feature performs conventional imagestabilization processing on the live video that the local user haschosen to send to the remote user, and vice versa. The imagestabilization feature thus minimizes the movement of objects that existin this live video resulting from subtle movement or shaking of thelocal mobile device while the live video is being captured. As such, theimage stabilization feature is also advantageous in the just mentionedsituation where a user is unable to keep their mobile devicesubstantially steady while the live video is being captured. It will beappreciated that it can be advantageous to combine the imagestabilization and object tracking features since if image stabilizationprocessing is first performed on the live video, it will be easier torecognize and track objects that exist therein.

1.6 Object Sharing Feature

This section describes an object sharing feature of the conferencingtechnique embodiments described herein. The object sharing featuregenerally allows a given user to share a desired digital object with theother user and collaboratively digitally annotate this object. Any typeof digital object can be shared, examples of which include a map, apicture, and a document, among others. More particularly, in anexemplary embodiment of the object sharing feature the local user candisplay a desired digital object on the entire local display screen,upon which the local mobile device will transmit the object over thenetwork to the remote mobile device. Upon receiving the digital objectover the network from the local mobile device, the remote mobile devicewill display the object on the entire remote display screen.Accordingly, both the local and remote users will be viewing the sameobject on their respective entire display screens and the users cancollaboratively digitally annotate the object as follows.

Whenever the local user makes a digital annotation on the digital objectthat is currently being displayed on the entire local display screen(such as when the local user digitally inks a route on a map that iscurrently being shared with the remote user), the local mobile devicewill display the annotation on top of the digital object that iscurrently being displayed on the entire local display screen, and willalso transmit the annotation over the network to the remote mobiledevice. Upon receiving this digital annotation over the network from thelocal mobile device, the remote mobile device will display the receivedannotation, without it being resized or repositioned, on top of thedigital object that is currently being displayed on the entire remotedisplay screen (e.g., the remote user can see the route that the localuser digitally inked on the map). Similarly, whenever the remote usermakes a digital annotation on the digital object that is currently beingdisplayed on the entire remote display screen, the remote mobile devicewill display the annotation on top of the digital object that iscurrently being displayed on the entire remote display screen, and willalso transmit the annotation over the network to the local mobiledevice. Upon receiving this digital annotation over the network from theremote mobile device, the local mobile device will display the receivedannotation, without it being resized or repositioned, on top of thedigital object that is currently being displayed on the entire localdisplay screen.

It is noted that any digital annotations that are made by the local orremote users can be anchored to the digital object that is currentlybeing displayed on the entire local and remote display screens. As aresult, whenever a given user zooms in on (i.e., magnifies) the digitalobject, any digital annotations that are currently being displayed ontop of the object will be magnified accordingly (e.g., the route thatthe local user digitally inked on the map appears correctly on thezoomed in map). Whenever the user zooms out on (i.e., shrinks/minifies)the digital object, any digital annotations that are currently beingdisplayed on top of the object will be minified accordingly. Wheneverthe user scrolls the digital object in a desired direction, any digitalannotations that are currently being displayed on top of the object willbe scrolled accordingly. It is also noted that the just described rolesof the local and remote users can be reversed. In other words, in analternate embodiment of the object sharing feature the remote user candisplay a desired digital object on the entire remote display screen,upon which the remote mobile device will transmit the object over thenetwork to the local mobile device. Upon receiving the digital objectover the network from the remote mobile device, the local mobile devicewill display the object on the entire local display screen. Both thelocal and remote users can then collaboratively digitally annotate thedigital object in the just described manner.

In a default embodiment of the object sharing feature the local mobiledevice transmits the live video that is currently being captured by thelocal front video capture device (e.g., the live video of the localuser) over the network to the remote mobile device, and upon receivingthis live video the remote mobile device resizes it as necessary to fitwithin the inset sector on the remote display screen, and then displaysthis resized live video within this inset sector. Similarly, the remotemobile device transmits the live video that is currently being capturedby the remote front video capture device (e.g, the live video of theremote user) over the network to the local mobile device, and uponreceiving this live video the local mobile device resizes it asnecessary to fit within the inset sector on the local display screen,and then displays this resized live video within this inset sector. Asdescribed heretofore, this allows each user to see live video of theother user while they are collaboratively discussing and annotating agiven digital object. Alternate embodiments of the object sharingfeature are also possible where, as described heretofore, a given usercan choose to send the live video that is currently being captured bytheir rear video capture device to the other user. In such a situationthe given user's mobile device will transmit the live video that iscurrently being captured by its rear video capture device over thenetwork to the other mobile device, and upon receiving this live videothe other mobile device will resize it as necessary and then displaythis resized live video within the inset sector on the other mobiledevice's display screen.

1.7 Application Sharing Feature

This section describes an application sharing feature of theconferencing technique embodiments described herein. The applicationsharing feature generally allows a given user to share a desiredapplication with the other user and collaboratively digitally annotatethis application. Any type of application can be shared, examples ofwhich include a web browser application, an interactive map application,and a photo gallery application, among others. More particularly, in anexemplary embodiment of the application sharing feature the local usercan open an application on the entire local display screen, upon whichthe local mobile device will transmit a message to run the sameapplication over the network to the remote mobile device. Upon receivingthis message over the network from the local mobile device, the remotemobile device will open the application on the entire remote displayscreen. Accordingly, both the local and remote users will be running andviewing the same application on their respective entire display screensand the users can collaboratively digitally annotate the application asfollows.

Whenever the local user makes a digital annotation on the applicationthat is currently being displayed on the entire local display screen(such as when the local user digitally inks a circle around a particularimage or information link that is currently being displayed in theapplication), the local mobile device will display the annotation on topof the application that is currently being displayed on the entire localdisplay screen, and will also transmit the annotation over the networkto the remote mobile device. Upon receiving this digital annotation overthe network from the local mobile device, the remote mobile device willdisplay the received annotation, without it being resized orrepositioned, on top of the application that is currently beingdisplayed on the entire remote display screen (e.g., the remote user cansee the circle that the local user digitally inked on the application).Similarly, whenever the remote user makes a digital annotation on theapplication that is currently being displayed on the entire remotedisplay screen, the remote mobile device will display the annotation ontop of the application that is currently being displayed on the entireremote display screen, and will also transmit the annotation over thenetwork to the local mobile device. Upon receiving this digitalannotation over the network from the remote mobile device, the localmobile device will display the received annotation, without it beingresized or repositioned, on top of the application that is currentlybeing displayed on the entire local display screen.

It is noted that any digital annotations that are made by the local orremote users can be anchored to the application that is currently beingdisplayed on the entire local and remote display screens. As a result,whenever a given user zooms in on (i.e., magnifies) the application, anydigital annotations that are currently being displayed on top of theapplication will be magnified accordingly (e.g., the circle that thelocal user digitally inked on the application appears correctly on thezoomed in application). Whenever the user zooms out on (i.e.,shrinks/minifies) the application, any digital annotations that arecurrently being displayed on top of the application will be minifiedaccordingly. Whenever the user scrolls the application in a desireddirection, any digital annotations that are currently being displayed ontop of the application will be scrolled accordingly. It is also notedthat the just described roles of the local and remote users can bereversed. In other words, in an alternate embodiment of the applicationsharing feature the remote user can open an application on the entireremote display screen, upon which the remote mobile device will transmita message to run the same application over the network to the localmobile device. Upon receiving this message over the network from theremote mobile device, the local mobile device will open the applicationon the entire local display screen. Both the local and remote users canthen collaboratively digitally annotate the application in the justdescribed manner.

In a default embodiment of the application sharing feature the localmobile device transmits the live video that is currently being capturedby the local front video capture device (e.g., the live video of thelocal user) over the network to the remote mobile device, and uponreceiving this live video the remote mobile device resizes it asnecessary to fit within the inset sector on the remote display screen,and then displays this resized live video within this inset sector.Similarly, the remote mobile device transmits the live video that iscurrently being captured by the remote front video capture device (e.g.,the live video of the remote user) over the network to the local mobiledevice, and upon receiving this live video the local mobile deviceresizes it as necessary to fit within the inset sector on the localdisplay screen, and then displays this resized live video within thisinset sector. As described heretofore, this allows each user to see livevideo of the other user while they are collaboratively discussing andannotating a given application. Alternate embodiments of the applicationsharing feature are also possible where, as described heretofore, agiven user can choose to send the live video that is currently beingcaptured by their rear video capture device to the other user. In such asituation the given user's mobile device will transmit the live videothat is currently being captured by its rear video capture device overthe network to the other mobile device, and upon receiving this livevideo the other mobile device will resize it as necessary and thendisplay this resized live video within the inset sector on the othermobile device's display screen.

Generally speaking, the conferencing technique embodiments describedherein allow a given user to choose whether or not they want to havetheir view of the application that is open and running on their mobiledevice be synchronized with the other user's view of the sameapplication that is open and running on the other user's mobile device.More particularly, the conferencing technique embodiments allow the userto choose between a WYSIWIS (what you see is what I see) applicationsharing mode (which can be thought of as a view-synchronized mode of theapplication sharing feature) and a WYSINWIS (what you see is not what Isee) application sharing mode (which can be thought of as anon-view-synchronized mode of the application sharing feature).

There are various reasons why the local and remote users may want to berunning the same application but not have the local and remote views ofthe application be synchronized. Some of these reasons areuser-dependent and some are device-dependent. One example of such auser-dependent reason is that the local and remote users may want toindependently move to different parts of the application. Anotherexample of such a user-dependent reason is that the local and remoteusers may want to independently view data in different representations.For instance, one user may want to view some data as a bar chart, whilethe other user wants to view the data as a line chart. Yet anotherexample of such a user-dependent reason is that a given user may want tosee the just the results of their local interactions with theapplication (i.e., they don't want to have their view of the applicationbe affected by the remote user's remote interactions with theapplication). An example of such a device-dependent reason is that thesizes and/or resolutions of the display screens on the local and remotemobile devices may be different (e.g., the screen of the local mobiledevice may be significantly smaller than the screen of the remote mobiledevice so that the local user may have to zoom in a lot to be able tosee a desired level of detail.

Whenever the local and remote mobile devices are operating in theWYSIWIS application sharing mode, the results of the local user'sinteractions with the application that is open on the local displayscreen are also displayed on the remote display screen, and the resultsof the remote user's interactions with the application that is open onthe remote display screen are also displayed on the local display screen(i.e., the local user's view of the application that is running on thelocal mobile device is synchronized with the remote user's view of thesame application that is running on the remote mobile device). Wheneverthe local and remote mobile devices are operating in the WYSINWISapplication sharing mode, the results of the local user's interactionswith the application that is open on the local display screen are notdisplayed on the remote display screen, and the results of the remoteuser's interactions with the application that is open on the remotedisplay screen are not displayed on the local display screen (i.e., thelocal user's view of the application that is running on the local mobiledevice is not synchronized with the remote user's view of the sameapplication that is running on the remote mobile device). A user is ableto switch between the WYSIWIS and WYSINWIS application sharing modes atany time.

1.8 Digital Annotation Modes and User Input Modalities

The conferencing technique embodiments described herein support variousdigital annotation modes including, but not limited to, a permanentannotation mode, an erasable annotation mode, and a transient annotationmode. It is noted that a given user can switch at will between thesedifferent digital annotation modes.

Whenever a given mobile device is operating in the permanent annotationmode, any digital annotations that are made on a given item of digitalinformation that is currently being displayed on the display screen ofthe mobile device will continue to be displayed on top of this item forthe duration of the video conference session. In other words and moregenerally speaking, in the permanent annotation mode any digitalannotations that are made cannot be deleted for the duration of thesession (i.e., the annotations are permanent).

Various implementations of the erasable annotation mode are possible,two of which will now be described in more detail. Whenever a givenmobile device is operating in a local-only erasable annotation mode, agiven digital annotation that is made by a given user can be deleted byjust the user himself (no other user can delete the annotation).Whenever the remote user deletes a given digital annotation that theymade, the annotation will be also be removed from the local displayscreen. Similarly, whenever the local user deletes a given digitalannotation that they made, the annotation will be also be removed fromthe remote display screen. Whenever a given mobile device is operatingin a global erasable annotation mode, a given user can delete anydigital annotation that is currently being displayed on the displayscreen of their mobile device, regardless of which user made theannotation. Whenever the remote user deletes a given digital annotationthat they made, the annotation will also be removed from the localdisplay screen. Similarly, whenever the local user deletes a givendigital annotation that they made, the annotation will also be removedfrom the remote display screen.

Whenever a given mobile device is operating in the transient annotationmode, any digital annotation that is made by either the local or remoteuser will fade out on a given display screen a prescribed period of time(e.g., five seconds) after the initial display of the annotation. In oneembodiment of the transient annotation mode the annotation fades outuniformly. In another embodiment of the transient annotation mode theannotation fades out non-uniformly (e.g., the different points on theannotation can fade out independently in succession, thus providing theeffect of a disappearing tail).

It will be appreciated that the collaborative digital annotation featureof the conferencing technique embodiments described herein isadvantageous in that it gives both the local and remote users aconvenient way to graphically communicate their thoughts/comments abouta given item of digital information (e.g., a live video, or a snapshotthereof, or a digital object, or an application) that is currently beingshared between them. A user can digitally annotate an item of digitalinformation that is currently being displayed on the display screen oftheir mobile device in various ways using various user input modalities,where the annotation can include either text, or one or morediagrams/sketches, or a combination thereof. By way of example but notlimitation, in one embodiment of the conferencing technique describedherein the user can annotate the item of digital information by using apointing device that physically contacts the mobile device's displayscreen in a region thereof where this item is being displayed. Wheneverthe user makes one or more strokes on the display screen on top of theitem of digital information using the pointing device, each of thestrokes is digitized thus creating a digital ink representation thereof(i.e., a digital representation which mimics the strokes), and thisdigitization is displayed on top of this item. The pointing device canbe either a pen/stylus that the user holds, or one or more of the user'sfingers, among other things. In another embodiment of the conferencingtechnique the user can annotate the item of digital information byplacing one or more fingers in close proximity to the display screen,thus creating a “finger shadow” on the screen, and then moving theirfingers in an appropriate manner to articulate the desired annotation.In yet another embodiment of the conferencing technique the user canannotate the item of digital information by using either a keyboard, ora joystick, among other types of user input devices.

1.9 Virtual Pointing Feature

This section describes a virtual pointing feature of the conferencingtechnique embodiments described herein. The virtual pointing featuregenerally allows a given user to remotely point at a specific locationon a given item of digital information (e.g., a live video, or asnapshot thereof, or a digital object, or an application) that iscurrently being shared with the other user. More particularly, in anexemplary embodiment of the virtual pointing feature the local user canuse any of the aforementioned user input modalities to point at aspecific location on an item of digital information that is currentlybeing displayed on both the local display screen and the remote displayscreen. Whenever the local user does this pointing, the local mobiledevice will detect the specific location on the item of digitalinformation that the local user is pointing at, and will then transmitdata that identifies this specific location over the network to theremote mobile device. Upon receiving this data over the network from thelocal mobile device, the remote mobile device uses the data to display apointer element (such as an arrow, or a cross-hair, or the like) on topof the item of digital information that is currently being displayed onthe remote display screen, where the pointer element points at the samespecific location on this item that is currently being pointing at bythe local user. Whenever the local user changes the specific location onthe item of digital information they are pointing at, the local mobiledevice will detect the change and will then transmit updated data thatidentifies this change over the network to the remote mobile device.Upon receiving this updated data over the network from the local mobiledevice, the remote mobile device will use the updated data to change thelocation of the pointer element that is currently being displayed on theremote display screen accordingly.

Similarly, the remote user can use any of the aforementioned user inputmodalities to point at a specific location on an item of digitalinformation that is currently being displayed on both the local displayscreen and the remote display screen. Whenever the remote user does thispointing, the remote mobile device will detect the specific location onthe item of digital information that the remote user is pointing at, andwill then transmit data that identifies this specific location over thenetwork to the local mobile device. Upon receiving this data over thenetwork from the remote mobile device, the local mobile device will usethe data to display a pointer element on top of the item of digitalinformation that is currently being displayed on the local displayscreen, where the pointer element points at the same specific locationon this item that is currently being pointing at by the remote user.Whenever the remote user changes the specific location on the item ofdigital information they are pointing at, the remote mobile device willdetect the change and will then transmit updated data that identifiesthis change over the network to the local mobile device. Upon receivingthis updated data over the network from the remote mobile device, thelocal mobile device will use the updated data to change the location ofthe pointer element that is currently being displayed on the localdisplay screen accordingly.

It is noted that the virtual pointing feature leaves no digital ink onthe item of digital information that is currently being shared betweenthe local and remote users. Whenever the local user stops pointing atthe specific location on the item of digital information that iscurrently being displayed on the local display screen, the remote mobiledevice will remove the pointer element from the remote display screen.Similarly, whenever the remote user stops pointing at the specificlocation on the item of digital information that is currently beingdisplayed on the remote display screen, the local mobile device willremove the pointer element from the local display screen. In anexemplary embodiment of the virtual pointing feature conventionalpointer trails can be implemented in order to enhance the visibility ofthe pointer element while it is being moved.

1.10 Display Heterogeneity Features

This section describes a screen resolution normalization feature and anindependent zoom feature of the conferencing technique embodimentsdescribed herein. As will be appreciated from the more detaileddescription that follows, these particular features are associated witha WYSINWIS operational mode of the conferencing technique embodimentsdescribed herein.

Whenever the resolution of the local display screen is different thanthe resolution of the remote display screen, the screen resolutionnormalization feature adjusts the resolution of any item of digitalinformation (e.g., a live video, or a snapshot thereof, or a digitalobject, or an application) and any digital annotations that are madethereon that the local mobile device receives from the remote mobiledevice to match the resolution of the local display screen. The screenresolution normalization feature thus makes any difference in screenresolution between the local and remote display screens transparent tothe local and remote users.

Whenever the size of the local display screen is smaller than the sizeof the remote display screen, or the resolution of the local displayscreen is lower than the resolution of the remote display screen, theindependent zoom feature allows the local user to zoom in on a desiredregion of a given item of digital information that is currently beingdisplayed on the local display screen, and is also currently beingdisplayed on the remote display screen, without this item being zoomedin on the remote display screen. The independent zoom feature thusprovides both the local and remote users with the ability to see thesame level of detail in the desired region of this item, and makes anydifference in screen size or resolution between the local and remotedisplay screens transparent to the local and remote users.

1.11 Recording, Replay and Remote Zoom Features

This section describes a recording feature, a replay feature, and aremote zoom feature of the conferencing technique embodiments describedherein.

The recording feature automatically records a given video conferencesession and indexes the session while it is being recorded. It will beappreciated that this indexing can be implemented in various ways. Byway of example but not limitation, in an exemplary embodiment of therecording feature the indexing of the session that is being recorded isbased on when either the local user, or the remote user, is eithertalking, or making digital annotations (in many cases these are theparts of the session that the users are most interested in).

The replay feature generally allows a user to replay a desired part of avideo conference session either while the session is currently takingplace, or after the session is over. More particularly, whenever a givenvideo conference session is currently underway and the session is beingrecorded and indexed, the replay feature allows both the local andremote users to pause the session, rewind the recorded session to adesired index point, and play the recorded session starting at thedesired index point. The user who initiated this playing of the recordedsession can choose a desired playback speed which can be either normalspeed, or a speed that is slower than normal speed, or another speedthat is faster than normal speed. The replay feature will replay theitem of digital information (e.g., a live video, or a snapshot thereof,or a digital object, or an application) that was being shared betweenthe local and remote users starting at the desired index point to whichthe session was rewound, and will also replay any digital annotationsthat were made on this item.

The replay feature supports two user-selectable modes, namely asynchronized replay mode and an independent replay mode. Whenever thelocal and remote mobile devices are operating in the synchronized replaymode and a given user plays the recorded session starting at a desiredindex point, this playing of the recorded session will be displayed onboth the entire local and entire remote display screens (i.e., both thelocal and remote users will see the replay at approximately the sametime). Whenever the local mobile device is operating in the independentreplay mode and the local user plays the recorded session starting at adesired index point, this playing of the recorded session is preventedfrom being displayed on the remote mobile device (i.e., the replay ofthe session will be displayed on just the local display screen).Similarly, whenever the remote mobile device is operating in theindependent replay mode and the remote user plays the recorded sessionstarting at a desired index point, this playing of the recorded sessionis prevented from being displayed on the local mobile device (i.e., thereplay of the session will be displayed on just the remote displayscreen). Whenever a video conference session is over but it was recordedand indexed, the replay feature allows any user to replay either theentire session or any desired part thereof.

The remote zoom feature generally allows a given user to remotely zoomeither in or out on the scene that is currently being captured by theother user's front or rear video capture device. More particularly,whenever the local user is currently sending the live video that iscurrently being captured by the local front video capture device to theremote user, the remote user can remotely control the local front videocapture device to zoom either in or out on the scene that is currentlybeing captured thereby. Whenever the local user is currently sending thelive video that is currently being captured by the local rear videocapture device to the remote user, the remote user can also remotelycontrol the local rear video capture device to zoom either in or out onthe scene that is currently being captured thereby. Similarly, wheneverthe remote user is currently sending the live video that is currentlybeing captured by the remote front video capture device to the localuser, the local user can remotely control the remote front video capturedevice to zoom either in or out on the scene that is currently beingcaptured thereby. Whenever the remote user is currently sending the livevideo that is currently being captured by the remote rear video capturedevice to the local user, the local user can also remotely control theremote rear video capture device to zoom either in or out on the scenethat is currently being captured thereby. In one embodiment of theremote zoom feature the zooming is done using a conventional digitalzoom method. In another embodiment of the remote zoom feature thezooming is done using a conventional optical zoom method.

1.12 Layering Options

This section describes various options for layering the different typesof information that can be displayed on the display screen of a givenmobile device.

In one embodiment of the conferencing technique described herein thecontents of the inset sector are displayed in a semi-transparent manneron the local display screen. In the aforementioned situations where thecontents of the inset sector include the live video that is currentlybeing captured by the remote front video capture device (e.g., livevideo of the remote user's head and torso), this particular embodimentcan give the illusion that the remote user is on the other side of thelocal display screen, immediately behind the digital information that isbeing shared between the users. In another embodiment of theconferencing technique the contents of the inset sector are displayed ina non-transparent manner on the local display screen.

Digital annotations that are made by a given user on a given item ofdigital information can be displayed on top of this item in variousways. By way of example but not limitation, in one embodiment of theconferencing technique described herein the digital annotations aredisplayed as an overlay layer on the item. In one implementation of thisembodiment the overlay layer is anchored to the item so that wheneverthe user zooms in on the item the annotations are magnified accordingly,whenever the user zooms out on the item the annotations are minifiedaccordingly, and whenever the user scrolls the item in a desireddirection the annotations are scrolled accordingly. In anotherimplementation of this embodiment the overlay layer is not anchored tothe item so that the digital annotations will not be magnified/minifiedwhenever the user zooms in/out on the item, and the annotations will notbe scrolled whenever the user scrolls the item. In another embodiment ofthe conferencing technique the digital annotations are displayed bysetting the color of the appropriate pixels in the item to a prescribedcolor (such as blue, among other colors). It will be appreciated thatthis embodiment also serves to anchor the annotations to the item sothat the annotations will be magnified/minified whenever the user zoomsin/out on the item, and the annotations will be scrolled whenever theuser scrolls the item.

1.13 Laser Pointing Feature

This section describes a laser pointing feature of the conferencingtechnique embodiments described herein. A given mobile device canoptionally include a laser pointer device that is generally pointed awayfrom the mobile device's user (i.e., the laser pointer device generallypoints in the same direction as the mobile device's rear video capturedevice). Various implementations of the laser pointer device arepossible. By way of example but not limitation, the laser pointer devicecan be either integrated into the rear video capture device, orseparately mounted on the rear of the mobile device. In one embodimentof the laser pointer feature the laser pointer device is a conventionalnon-steerable laser that points in a fixed direction which issubstantially aligned with the axis of the rear video capture device. Inanother embodiment of the laser pointer feature the laser pointer deviceis a conventional steerable laser (e.g., a laser combined with a MEMs(micro-electro-mechanical systems) mirror) that points in a non-fixeddirection which is controllable (i.e., steerable) by both the local andremote users.

Whenever the local mobile device includes a local laser pointer devicethe laser pointing feature of the conferencing technique embodimentsdescribed herein allows both the local and remote users to use the locallaser pointer device to illuminate a real object of interest (i.e.,“paint” the real object) in the scene that is substantially in front ofthe local user (and thus is currently being captured by the local rearvideo capture device) with a visible spot of light having a prescribedcolor (e.g., red, among other colors). It will be appreciated that thisspot will be visible in the live video of this scene that is currentlybeing captured by the local rear video capture device, and thus willalso be visible to the remote user. Therefore, the laser pointingfeature allows both the local and remote users to visuallyhighlight/pinpoint a real object of interest in the scene that issubstantially in front of the local user. Both the local and remoteusers will then see the highlighted real object, and can collaborativelydiscuss and digitally annotate it.

The aforementioned object tracking feature can be used to recognize andtrack the real object of interest in the scene upon which the visiblespot of light is being illuminated, and then maintain this spot of lighton this real object in the live video of the scene that is currentlybeing captured and transmitted over the network. It will thus beappreciated that combining the laser pointing and object trackingfeatures is advantageous whenever the local user is moving the localmobile device around the real object while the local laser pointerdevice is being used to illuminate the real object with the spot oflight. Combining the laser pointing and object tracking features is alsoadvantageous in another situation where slight movements of the localmobile device occur due to the local user's inability to keep the localmobile device perfectly steady while they are holding it and using thelocal laser pointer device to illuminate the real object with the smallbright spot of light.

1.14 Process Framework

FIG. 4 illustrates one embodiment, in simplified form, of a process forallowing the local user of the local mobile device to participate in avideo conference session with the remote user of the remote mobiledevice. As exemplified in FIG. 4, the process starts in block 400 withreceiving live video over the network from the remote mobile device. Thereceived live video is then displayed on the entire local display screen(block 402). Upon the local user selecting either the live video of thelocal user (that is captured by the local front video capture device),or the live video of the scene that is in front of the local user (thatis captured by the local rear video capture device), the selected livevideo is transmitted over the network to the remote mobile device (block404). The selected live video is then resized as necessary to fit withinthe inset sector on the local display screen (block 406). The resizedselected live video is then displayed within the inset sector (block408). Whenever the local user makes a first digital annotation on thereceived live video (block 410, Yes), the first digital annotation isdisplayed on top of the received live video (block 412), and the firstdigital annotation is transmitted over the network to the remote mobiledevice (block 414). Whenever the remote user makes a second digitalannotation on the selected live video (block 416, Yes), the seconddigital annotation is received over the network from the remote mobiledevice (block 418), the received second digital annotation is resized asnecessary to fit within the inset sector (block 420), and the resizedreceived second digital annotation is displayed on top of the resizedselected live video (block 422).

FIG. 5 illustrates another embodiment, in simplified form, of a processfor allowing the local user of the local mobile device to participate ina video conference session with the remote user of the remote mobiledevice. As exemplified in FIG. 5, the process starts in block 500 withtransmitting the live video of the local user over the network to theremote mobile device. Live video of the remote user is then receivedover the network from the remote mobile device (block 502). The receivedlive video is then resized as necessary to fit within the inset sectoron the local display screen (block 504). The resized received live videois then displayed within the inset sector (block 506). Upon the localuser opening an application on the entire local display screen, amessage to run the application is transmitted over the network to theremote mobile device (block 508). Whenever the local user makes a firstdigital annotation on the application (block 510, Yes), the firstdigital annotation is displayed on top of the application (block 512),and the first digital annotation is transmitted over the network to theremote mobile device (block 514). Whenever the remote user makes asecond digital annotation on the application (block 516, Yes), thesecond digital annotation is received over the network from the remotemobile device (block 518), and the received second digital annotation isdisplayed on top of the application (block 520).

FIG. 6 illustrates yet another embodiment, in simplified form, of aprocess for allowing the local user of the local mobile device toparticipate in a video conference session with the remote user of theremote mobile device. As exemplified in FIG. 6, the process starts inblock 600 with transmitting the live video of the local user over thenetwork to the remote mobile device. Live video of the remote user isthen received over the network from the remote mobile device (block602). The received live video is then resized as necessary to fit withinthe inset sector on the local display screen (block 604). The resizedreceived live video is then displayed within the inset sector (block606). Upon the local user displaying a digital object on the entirelocal display screen, the object is transmitted over the network to theremote mobile device (block 608). Whenever the local user makes a firstdigital annotation on the object (block 610, Yes), the first digitalannotation is displayed on top of the object (block 612), and the firstdigital annotation is transmitted over the network to the remote mobiledevice (block 614). Whenever the remote user makes a second digitalannotation on the object (block 616, Yes), the second digital annotationis received over the network from the remote mobile device (block 618),and the received second digital annotation is displayed on top of theobject (block 620).

2.0 Additional Embodiments

While the conferencing technique has been described by specificreference to embodiments thereof, it is understood that variations andmodifications thereof can be made without departing from the true spiritand scope of the conferencing technique. By way of example but notlimitation and with regard to the video snapshot feature describedherein, in addition to the local user being able to take a snapshot ofthe live video they have chosen to send to the remote user, the localuser can also take a snapshot of the live video they are currentlyreceiving from the remote user.

Furthermore, although the conferencing technique embodiments have beendescribed herein in the context of a local user participating in a videoconference session with a remote user, the conferencing techniqueembodiments are also operational in situations where three or more users(i.e., three or more mobile devices) are participating in a videoconference session. Yet furthermore, although the conferencing techniqueembodiments have been described in the context of each of the usersusing a mobile device, the conferencing technique embodiments are alsooperational in situations where one or more of the users is using anetworking-enabled client computing device that is larger than a mobiledevice and thus is generally not handheld (such as a desktop personalcomputer (PC), or a laptop computer, or the like), includes a frontvideo capture device that is pointed toward the user, and may also beconfigured to include one or more of a rear video capture device that ispointed away from the user, or a touch-sensitive display screen. Yetfurthermore, although the conferencing technique embodiments have beendescribed in the context of a peer-to-peer framework where the mobiledevices communicate directly with each other, the conferencing techniqueembodiments are also operational in a client-server framework where themobile devices communicate with each other through a shared server.

Additionally, a video freeze-frame alternative to the video snapshotfeature is possible where, rather than taking a snapshot of a given livevideo, the local user can freeze the live video to a specific frame.Upon the local user freezing the live video to a specific frame, thelocal mobile device will display the frozen frame on the entire localdisplay screen, and will also transmit the frozen frame over the networkto the remote mobile device. Upon receiving this frozen frame, theremote mobile device will display it on the entire remote displayscreen. Accordingly, both the local and remote users will be viewing thesame frozen frame on their respective display screens and the users cancollaboratively digitally annotate this frozen frame as describedheretofore. The local user can subsequently “unfreeze” the live video,at which point the local and remote mobile devices will resume the videoconferencing state they were in just before the local user invoked thefreeze.

It is also noted that any or all of the aforementioned embodiments canbe used in any combination desired to form additional hybridembodiments. Although the conferencing technique embodiments have beendescribed in language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features or acts described heretofore. Rather, the specificfeatures and acts described heretofore are disclosed as example forms ofimplementing the claims.

3.0 Computing Environment

The conferencing technique embodiments described herein are operationalwithin numerous types of general purpose or special purpose computingsystem environments or configurations. FIG. 7 illustrates a simplifiedexample of a general-purpose computer system on which variousembodiments and elements of the conferencing technique, as describedherein, may be implemented. It is noted that any boxes that arerepresented by broken or dashed lines in FIG. 7 represent alternateembodiments of the simplified computing device, and that any or all ofthese alternate embodiments, as described below, may be used incombination with other alternate embodiments that are describedthroughout this document.

For example, FIG. 7 shows a general system diagram showing a simplifiedcomputing device 700. Such computing devices can be typically be foundin devices having at least some minimum computational capability,including, but not limited to, personal computers (PCs), servercomputers, handheld computing devices, laptop or mobile computers,communications devices such as cell phones and personal digitalassistants (PDAs), multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, and audio or video media players.

To allow a device to implement the conferencing technique embodimentsdescribed herein, the device should have a sufficient computationalcapability and system memory to enable basic computational operations.In particular, as illustrated by FIG. 7, the computational capability isgenerally illustrated by one or more processing unit(s) 710, and mayalso include one or more graphics processing units (GPUs) 715, either orboth in communication with system memory 720. Note that that theprocessing unit(s) 710 may be specialized microprocessors (such as adigital signal processor (DSP), a very long instruction word (VLIW)processor, or other micro-controller) or can be conventional centralprocessing units (CPUs) having one or more processing cores including,but not limited to, specialized GPU-based cores in a multi-core CPU.

In addition, the simplified computing device 700 of FIG. 7 may alsoinclude other components, such as, for example, a communicationsinterface 730. The simplified computing device 700 of FIG. 7 may alsoinclude one or more conventional computer input devices 740 (e.g.,pointing devices, keyboards, audio (e.g., voice) input/capture devices,video input/capture devices, haptic input devices, devices for receivingwired or wireless data transmissions, and the like). The simplifiedcomputing device 700 of FIG. 7 may also include other optionalcomponents, such as, for example, one or more conventional computeroutput devices 750 (e.g., display device(s) 755, audio output devices,video output devices, devices for transmitting wired or wireless datatransmissions, and the like). Note that typical communicationsinterfaces 730, input devices 740, output devices 750, and storagedevices 760 for general-purpose computers are well known to thoseskilled in the art, and will not be described in detail herein.

The simplified computing device 700 of FIG. 7 may also include a varietyof computer readable media. Computer readable media can be any availablemedia that can be accessed by the computer 700 via storage devices 760,and includes both volatile and nonvolatile media that is eitherremovable 770 and/or non-removable 780, for storage of information suchas computer-readable or computer-executable instructions, datastructures, program modules, or other data. By way of example but notlimitation, computer readable media may include computer storage mediaand communication media. Computer storage media includes, but is notlimited to, computer or machine readable media or storage devices suchas digital versatile disks (DVDs), compact discs (CDs), floppy disks,tape drives, hard drives, optical drives, solid state memory devices,random access memory (RAM), read-only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), flash memory or othermemory technology, magnetic cassettes, magnetic tapes, magnetic diskstorage, or other magnetic storage devices, or any other device whichcan be used to store the desired information and which can be accessedby one or more computing devices.

Storage of information such as computer-readable or computer-executableinstructions, data structures, program modules, and the like, can alsobe accomplished by using any of a variety of the aforementionedcommunication media to encode one or more modulated data signals orcarrier waves, or other transport mechanisms or communicationsprotocols, and includes any wired or wireless information deliverymechanism. Note that the terms “modulated data signal” or “carrier wave”generally refer to a signal that has one or more of its characteristicsset or changed in such a manner as to encode information in the signal.For example, communication media includes wired media such as a wirednetwork or direct-wired connection carrying one or more modulated datasignals, and wireless media such as acoustic, radio frequency (RF),infrared, laser, and other wireless media for transmitting and/orreceiving one or more modulated data signals or carrier waves.Combinations of the any of the above should also be included within thescope of communication media.

Furthermore, software, programs, and/or computer program productsembodying the some or all of the various embodiments of the conferencingtechnique described herein, or portions thereof, may be stored,received, transmitted, or read from any desired combination of computeror machine readable media or storage devices and communication media inthe form of computer executable instructions or other data structures.

Finally, the conferencing technique embodiments described herein may befurther described in the general context of computer-executableinstructions, such as program modules, being executed by a computingdevice. Generally, program modules include routines, programs, objects,components, data structures, and the like, that perform particular tasksor implement particular abstract data types. The conferencing techniqueembodiments may also be practiced in distributed computing environmentswhere tasks are performed by one or more remote processing devices, orwithin a cloud of one or more devices, that are linked through one ormore communications networks. In a distributed computing environment,program modules may be located in both local and remote computer storagemedia including media storage devices. Additionally, the aforementionedinstructions may be implemented, in part or in whole, as hardware logiccircuits, which may or may not include a processor.

Wherefore, what is claimed is:
 1. A computer-implemented process forallowing a local user of a local mobile device to participate in a videoconference session with a remote user of a remote mobile device, whereinthe local and remote mobile devices are interconnected by a network, andthe local mobile device comprises a local display screen, a front videocapture device that is pointed toward the local user and captures livevideo of the local user, and a rear video capture device that is pointedaway from the local user and captures live video of a scene that is infront of the local user, comprising: using the local mobile device toperform the following process actions: receiving live video over thenetwork from the remote mobile device; displaying the received livevideo on the local display screen; upon the local user selecting eitherthe live video of the local user, or the live video of the scene that isin front of the local user, transmitting the selected live video overthe network to the remote mobile device; resizing the selected livevideo as necessary to fit within an inset sector on the local displayscreen; and displaying the resized selected live video within the insetsector; and whenever the local user makes a first digital annotation onthe received live video, displaying the first digital annotation on topof the received live video, recognizing and tracking objects that existin the received live video, and anchoring the first digital annotationto said objects, wherein said anchoring allows the first digitalannotation to track said objects and maintain its orientation relativeto said objects if said objects move.
 2. The process of claim 1, furthercomprising the actions of: whenever the local user makes a first digitalannotation on the received live video, transmitting the first digitalannotation over the network to the remote mobile device; whenever theremote user makes a second digital annotation on the selected livevideo, receiving the second digital annotation over the network from theremote mobile device, resizing the received second digital annotation asnecessary to fit within the inset sector, and displaying the resizedreceived second digital annotation on top of the resized selected livevideo; whenever the local user makes a third digital annotation on theresized selected live video, displaying the third digital annotation ontop of the resized selected live video, and transmitting the thirddigital annotation over the network to the remote mobile device; andwhenever the remote user makes a fourth digital annotation on thereceived live video, receiving the fourth digital annotation over thenetwork from the remote mobile device, resizing the received fourthdigital annotation as necessary to fit on the entire local displayscreen, and displaying the resized received fourth digital annotation ontop of the received live video.
 3. The process of claim 2, furthercomprising the actions of: time correlating the first digital annotationto the particular frames of the received live video upon which the firstdigital annotation is made in order to account for time lag and anyjitter that is present in the network; and time correlating the thirddigital annotation to the particular frames of the resized selected livevideo upon which the third digital annotation is made in order toaccount for said time lag and jitter.
 4. The process of claim 1, furthercomprising the actions of: whenever the local user repositions the insetsector on the local display screen, repositioning the resized selectedlive video and the resized received second digital annotationaccordingly; and whenever the local user resizes the inset sector,resizing the resized selected live video and the resized received seconddigital annotation accordingly.
 5. The process of claim 1, furthercomprising the actions of: allowing the local user to choose which livevideo is displayed on the entire local display screen; and whenever thelocal user chooses to display the selected live video on the entirelocal display screen, displaying the selected live video on the entirelocal display screen.
 6. The process of claim 1, further comprising theactions of: whenever the local user takes a snapshot of the selectedlive video, displaying said snapshot on the entire local display screen,and transmitting said snapshot over the network to the remote mobiledevice; whenever the local user makes a local digital annotation on saidsnapshot, displaying the local digital annotation on top of saidsnapshot, and transmitting the local digital annotation over the networkto the remote mobile device; and whenever the remote user makes a remotedigital annotation on said snapshot, receiving the remote digitalannotation over the network from the remote mobile device, anddisplaying the received remote digital annotation on top of saidsnapshot.
 7. The process of claim 6, further comprising an action ofanchoring the local and remote digital annotations to the snapshot sothat, whenever the local user zooms in on the snapshot said annotationsare magnified accordingly, whenever the local user zooms out on thesnapshot said annotations are minified accordingly, and whenever thelocal user scrolls the snapshot in a desired direction said annotationsare scrolled accordingly.
 8. The process of claim 6, further comprisingthe actions of: whenever the local user points at a first location onthe snapshot, detecting said first location, and transmitting dataidentifying said first location over the network to the remote mobiledevice; and whenever the remote user points at a second location on thesnapshot, receiving data identifying said second location over thenetwork from the remote mobile device, and using said received data todisplay a pointer element on top of the snapshot, wherein the pointerelement points at the second location on the snapshot.
 9. The process ofclaim 1, further comprising an action of performing image stabilizationprocessing on the selected live video.
 10. The process of claim 1,further comprising the actions of: whenever the local user makes a firstdigital annotation on the received live video transmitting the firstdigital annotation over the network to the remote mobile device;whenever the remote user makes a second digital annotation on theselected live video, receiving the second digital annotation over thenetwork from the remote mobile device, resizing the received seconddigital annotation as necessary to fit within the inset sector, anddisplaying the resized received second digital annotation on top of theresized selected live video; whenever the local mobile device isoperating in a permanent annotation mode, continuing to display thefirst digital annotation on top of the received live video for theduration of the session, and continuing to display the resized receivedsecond digital annotation on top of the resized selected live video forthe duration of the session; whenever the local mobile device isoperating in a local-only erasable annotation mode, allowing just thelocal user to delete the first digital annotation, allowing just theremote user to delete the second digital annotation, and whenever theremote user deletes the second digital annotation, removing the resizedreceived second digital annotation from the local display screen;whenever the local mobile device is operating in a global erasableannotation mode, allowing the local user to delete either the firstdigital annotation or the resized received second digital annotation,and whenever the remote user deletes the second digital annotation,removing the resized received second digital annotation from the localdisplay screen; and whenever the local mobile device is operating in atransient annotation mode, fading out the first digital annotation beingdisplayed on top of the received live video a prescribed period of timeafter the initial display of the first digital annotation, and fadingout the resized received second digital annotation being displayed ontop of the resized selected live video the prescribed period of timeafter the initial display of the resized received second digitalannotation.
 11. The process of claim 1, wherein the remote mobile devicecomprises a remote display screen, further comprising an action of,whenever the resolution of the local display screen is different thanthe resolution of the remote display screen, adjusting the resolution ofthe received live video to match the resolution of the local displayscreen.
 12. The process of claim 1, wherein the local mobile devicefurther comprises a laser pointer device that is pointed away from thelocal user, further comprising an action of allowing both the local andremote users to use the laser pointer device to illuminate a real objectof interest in the scene that is in front of the local user with avisible spot of light, wherein said spot is visible in the live video ofsaid scene and is thus is visible to the remote user.
 13. The process ofclaim 12, wherein either, the laser pointer device comprises anon-steerable laser that points in a fixed direction which is alignedwith the axis of the rear video capture device, or the laser pointerdevice comprises a steerable laser that points in a direction which iscontrollable by both the local and remote users.
 14. The process ofclaim 1, further comprising the actions of: recording the session; andindexing the session while it is being recorded, wherein said indexingis based on when either the local user, or the remote user, is eithertalking, or making digital annotations.
 15. The process of claim 14,further comprising the actions of: allowing both the local and remoteusers to pause the session, rewind the recorded session to a desiredindex point, and play the recorded session starting at the desired indexpoint; whenever the local mobile device is operating in a synchronizedreplay mode and the remote user plays the recorded session starting at afirst index point, displaying said playing of the recorded sessionstarting at the first index point on the entire local display screen;and whenever the local mobile device is operating in an independentreplay mode and the local user plays the recorded session starting at asecond index point, preventing said playing of the recorded sessionstarting at the second index point from being displayed on the remotemobile device.
 16. A computer-implemented process for allowing a localuser of a local mobile device to participate in a video conferencesession with a remote user of a remote mobile device, wherein the localand remote mobile devices are interconnected by a network, and the localmobile device comprises a display screen and a front video capturedevice that is pointed toward the local user and captures live video ofthe local user, comprising: using the local mobile device to perform thefollowing process actions: transmitting the live video of the local userover the network to the remote mobile device; receiving live video ofthe remote user over the network from the remote mobile device; resizingthe received live video as necessary to fit within an inset sector onthe display screen; displaying the resized received live video withinthe inset sector; upon the local user opening an application on theentice display screen, transmitting a message to run the applicationover the network to the remote mobile device; whenever the local usermakes a first digital annotation on the application, displaying thefirst digital annotation on top of the application, and transmitting thefirst digital annotation over the network to the remote mobile device;and whenever the remote user makes a second digital annotation on theapplication, receiving the second digital annotation over the networkfrom the remote mobile device, and displaying the received seconddigital annotation on top of the application.
 17. The process of claim16, further comprising an action of, whenever the local mobile device isoperating in a what you see is what I see application sharing mode,synchronizing the view of the application that is open on the entiredisplay screen with the view of the application that is being run on theremote mobile device.
 18. The process of claim 16, further comprising anaction of anchoring the first and second digital annotations to theapplication so that, whenever the local user zooms in on the applicationsaid annotations are magnified accordingly, whenever the local userzooms out on the application said annotations are minified accordingly,and whenever the local user scrolls the application in a desireddirection said annotations are scrolled accordingly, wherein saidanchoring comprises either displaying said annotations as an overlaylayer on the application wherein said layer is anchored to theapplication, or displaying said annotations by setting the color ofappropriate pixels in the application to a prescribed color.
 19. Acomputer-implemented process for allowing a local user of a local mobiledevice to participate in a video conference session with a remote userof a remote mobile device, wherein the local and remote mobile devicesare interconnected by a network, and the local mobile device comprises adisplay screen and a front video capture device that is pointed towardthe local user and captures live video of the local user, comprising:using the local mobile device to perform the following process actions:transmitting the live video of the local user over the network to theremote mobile device; receiving live video of the remote user over thenetwork from the remote mobile device; resizing the received live videoas necessary to fit within an inset sector on the display screen;displaying the resized received live video within the inset sector uponthe local user displaying a digital object on the entire display screen,transmitting said object over the network to the remote mobile device;whenever the local user makes a first digital annotation on said object,displaying the first digital annotation on top of said object, andtransmitting the first digital annotation over the network to the remotemobile device; and whenever the remote user makes a second digitalannotation on said object, receiving the second digital annotation overthe network from the remote mobile device, and displaying the receivedsecond digital annotation on top of said object.